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We study the quasiparticle interference (QPI) patterns caused by scattering off nonmagnetic, 
magnetic point impurities, and edge impurities, separately, in a two dimensional helical liquid, 
which describes the surface states of a topological insulator. The unique features associated with 
hexagonal warping effects are identified in the QPI patterns of charge density with nonmagnetic 
impurities and spin density with magnetic impurities. The symmetry properties of the QPI patterns 
can be used to determine the symmetry of microscopic models. The Friedel oscillation is calculated 
for edge impurities and the decay of the oscillation is not universal, strongly depending on Fermi 
energy. Some discrepancies between our theoretical results and current experimental observations 
are discussed. 



I. INTRODUCTION 

Several recent theoretical^^ and 

experimental^&£&£ works have focused on a new 
quantum state of matter, topological insulators in three 
dimensions, which exhibit bulk insulating gaps (mainly 
of spin-orbit origin) while possess time-reversal symme- 
try protected gapless surface states. One of intriguing 
properties in this new quantum state comes from those 
"protected" surface states, which provide a lab-realizable 
condensed-matter analog of two dimensional, massless 
Dirac theory with "odd" number of species (Dirac 
cones), in the surface Brillouin zone (SBZ)iii£. The 
charge carriers on the surfaces here, the so-called 
(spin) helical Dirac fermions&ii, behave like relativistic 
particles with a spin locked to its momentum leading to 
the breakdown of the spin SU{2) rotational symmetry. 
This feature is sharply in contrast to graphene, where 
the system not only possesses an even number of Dirac 
cones in its spectrum, but the role of the "locked" spin 
is also replaced by a pseudo-spin (sublattice symmetry) 
and hence each Dirac cone still has two-fold spin 
degeneracy^. 

As a useful surface probe, recent angle-resolved pho- 
toemission spectroscopy (ARPES) experiments success- 
fully demonstrated the surface band structures with odd 
number of Dirac cones^i as well as the correspond- 
ing spin helical structures near a Dirac point£&£. Al- 
though the confirmed nature of the bands by ARPES 
suggests the quantum state to be topologically insulat- 
ing, the quest for new quantum phenomena uniquely as- 
sociated with such topology-protected surface states re- 
mains urgent and necessary. The usual way in solid state 
physics to explore the nontrivial electronic properties of 
helical Dirac fermion systems would be the transport 
measurement on the surface of a topological insulator—. 
However, such a measurement may not be practically 
straightforward, since (i) tuning the system to the topo- 
logical transport regime where the charge density van- 
ishes is tricky, and (ii) the presence of the n-type doping 
from vacancy (or anti-site defects) as well as the fact that 
the surface states surround the sample make the results 
difficult to be distinguished from the bulk and surface 



contributions^^. 

Alternatively, the quasiparticle interference (QPI) 
caused by scattering off impurities on a surface can pro- 
vide a way of revealing the topological nature of the sur- 
face state a 14 i 15 i 16 i 17 . The concept of QPI is elementary 
in quantum mechanics. For instance, due to impurity 
(elastic) scattering, the interference between the incom- 
ing and outgoing waves with momenta Iq and kj, respec- 
tively, can give rise to an amplitude modulation in the 
local density of states (LDOS) at wavevector q = k/ — k^. 
Such kind of interference pattern can be observed in 
Fourier transform scanning tunneling spectroscopy (FT- 
STS) nowadays and it has been proved useful in deter- 
mining the pairing nature of high-T c cuprates 18 . By mea- 
suring the QPI patterns and analyzing them through 
a convolution of ARPES data together with a spin- 
dependent scattering matrix element, Roushan and et 
ali^ were able to demonstrate the absence of backscat- 
tering in the topological surface states of Bii_ x Sb x ' , a 
key property of helical spin liquid. 

Most recently, based on symmetry analysis, a new 
hexagonal warping term, which is absent in Bi\- x Sb x > , 
is suggested by Liang Fui£ to explain the evolution of the 
Fermi surface of the effective 2D helical Dirac model de- 
scribing the surface band structure of a family of 3D topo- 
logical insulators, B12X3 (X=Se or Te). As measured in 
ARPES experiments, the shape of the Fermi surface (FS) 
evolves gradually from a hexagram, a hexagon, to a circle 
of shrinking volume, and finally meets at the Dirac point 
when lowering the Fermi energy. The new term leads to 
strong density variation around Fermi surface and also 
modifies the spin helical configuration. As a result, the 
existence of the new term can strongly modify the QPI. 
In other words, the QPI can provide a direct evidence to 
justify the model. 

In this paper we systematically investigate the inter- 
ference effects of a point-impurity and an edge-impurity 
scattering, respectively, on the LDOS in a 2D helical 
Dirac fermion system. We use T-matrix approach to cal- 
culate QPI spectra at a few representative energies, for 
emphasizing the effects of the hexagonal warping term, 
in the presence of a nonmagnetic / magnetic impurity. We 
also investigate an edge impurity by using a method gen- 
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eralized from ID scattering problems with a potential 
barrier. Several profound features are found in this study. 
In a nutshell, we observe: (i) the backward scattering by 
nonmagnetic point impurities is topologically suppressed, 
just as what has been shown in 14 with a simpler empir- 
ical analysis, and the dominant interference pattern be- 
comes that of spatial period 2tt/ |q[35 1 when going away 
from the Dirac regime (see Fig. |4] for the definition of 
q 35 ); (ii) In the presence of magnetic impurity, the QPI 
of charge density is very weak while that of spin density 
becomes strong. Near the Dirac regime, spin moments of 
fermions are flipped when scattering wave vector crosses 
over |q| = 2|kp|, as demonstrated in the (z-component) 
spin LDOS [see Fig. 0(b)]; (hi) the mirror symmetries 
of the spin LDOS in the presence of in-plane magnetic 
impurity with spin polarization fixed along x and y di- 
rections can be used to determine the symmetry of mi- 
croscopic models and to verify the presence or absence of 
the warping term; (iv) In the case of ID edge impurities, 
the Friedel oscillation has no universal decaying function. 
Depending on Fermi surface energy, we show that the os- 
cillation decays as l/^/\x\ if the FS shape is dominated 
by the warping term, and as |x| -3 / 2 if the warping term 
is negligible. These special quantum phenomena, sharply 
in contrast to conventional metals, are mainly associated 
with the 2D helical liquid. 
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FIG. 1: Contours of constant energy and the evolution of FS. 



II. THE MODEL AND T-MATRIX FORMALISM 

We now briefly introduce our used formalism below. 
The explicit model we study here is written as 

k 2 A 

H(k) = v(k x a y - k y (j x ) + — + -{k% + ki)a z , (1) 

where k± = k y ± ik x . v and A denote Fermi veloc- 
ity and hexagonal warping parameter, respectively. The 
Pauli matrices, c^, act on spin space of fermionic quasi- 
particles. The form of H(k) is suitable for describing 



the [111] surface band structure near T point in SBZ 
of a 3D topological insulator 1^2X3, and is fixed under 
general symmetry considerations, namely, time reversal 
and Cs v symmetries^. Notice that we have chosen x 
direction to be along TM in SBZ. The /c-linear term, 
Ho = v(k x a y — ky(j x ), describes an isotropic 2D helical 
Dirac fermions, and the /c-square term causes particle- 
hole asymmetry. More importantly, the k-cuhe warping 
term, H w = §(£rj_ + k 3 _)(j zi leads to hexagonal distortion 
of the Fermi surface. The resulting two energy bands 
now touch at the Dirac point (i.e., T point in SBZ) with 
dispersion relation, 

e± (k) = &* ± ]/v^ + ^(ki + kiy. (2) 

Defining the characteristic length scale b = yj X/v and en- 
ergy E* = v/b introduced by the hexagonal warping pa- 
rameter, we draw the contours of constant energy (CCE) 
in momentum space in units of 1/b and single-particle 
density of states (DOS) of H(k), respectively, in Figs. [1] 
and El 
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FIG. 2: Density of states based on the model in Eq. (pQ). 



In the numerical evaluation, we have taken b = 1 
, v = 0.25, and A = 0.25 such that the Fermi sur- 
face in 0.67% Sn-doped Bi2Te3 can be qualitatively re- 
produced, where the measured v = 2.55eV • A and 
Ep = 1.2E* ^0.3eV. Unless otherwise stated, we will 
assume particle-hole symmetry, i.e., m* — > 00. As shown 
in Figs. [T]and El when uj <C 0.2 the DOS is almost linear 
in co with more circular FS, while when uo ^> 0.2 the DOS 
behaves like a; -1 / 3 with hexagram-like FS. 

In addition to the CCE, we also present the spin- 
resolved FS with two representative energies used 
through out this paper, E D =0.05eV (0.2£?*) and 
E w =0.3eV (1.2£*) in Fig. EH They clearly demonstrate 
the "spin-helical" nature of the 2D fermions, which is 
indeed essential when analyzing the QPI spectra later. 
In particular, as uo = Ew, non- vanishing spin moments 
along z direction (out of surface plane) are present mainly 
due to <j z in the warping term, which is directly propor- 
tional to electron's spin. Notice that the spin moment 
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must be in-plane along TM (i.e., at each sharp vertex of 
the FS), which is a consequence of the odd parity of a z 
under the mirror operation y —> —y. 
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FIG. 3: Spin textures around the Fermi surface at uj = 0.3eV 
in (a) and at uj = 0.05eV (b). 

Next, we consider the quasiparticle scattering prob- 
lem within the T-matrix approach^. For a general 
AMmpurity problem, the impurity-induced electronic 
Green's function is given by 

N 

5G(t,y',u))= ^2 G (r,r i ,uj)T(r i: r j ,uj)Go(r j ,r / ,uj), 

(3) 

where the T-matrix obeys the Bethe-Salpeter equation 

N 

T(ri,Tj,(j) = V Ti 5 r . irj + V ri ^G (r i ,r fe ,cj)T(r /c ,r j ,cj), 

k=l 

(4) 

and the Green's function (in momentum space) of the 
clean system is 

G (k,Lj) = [u + iri-H(k)]- 1 . (5) 

In the case of a single point nonmagnetic (magnetic) 
impurity located at the origin, the scattering potential is 
simply V r = 5 r ,oVivj0"o (^.oVm/^)) where ctq is a 2 x 2 



identity matrix. Taking the advantages of the trans- 
lational symmetry of the clean system and momentum 
independence of the scattering potential (for instance, 
Vk,k' = Vni&o/N = V in the nonmagnetic case), one 
can simplify the formula as 

T(u) = [l-V [ ? ^Go(k,a;)]- 1 y, (6) 

A+(k)<A (2tt) 2 

and hence around the impurity, spatial oscillations of the 
local density of states are induced. To see the interference 
effects due to impurity scattering, it is more convenient to 
compute the Fourier-transformed (induced) local density 
of states (FT-LDOS), 

J d 2 re i ^ r 5p(v,uj) ~ Spfau) 

y e+ (k)<A ( 27r ) 

where #(k,q,tj) = Ei=# G ^( k > k + Q,^) ~ ^*;( k + 
q, k, uj)]. In general, p(q, uj) is a complex number. If we 
separately define the symmetric and antisymmetric parts 
of the LDOS as p s (x : y, uj) = \p(x, y, uj)-\-p(—x, —y, uj)]/2 
and p A (x,y,uj) = [p(x,y,uj) — p(—x,—y,uj)]/2, the real 
and imaginary parts of p(q, uj) simply describe the sym- 
metric and antisymmetric parts of the LDOS respec- 
tively. In the following discussion of the effects of non- 
magnetic impurities, since the real part is at least two 
orders of magnitude larger than the imaginary part, we 
focus on the former. In our calculation, we have intro- 
duced an energy cutoff A = AE* when integrating over 
momentum. Our main results do not sensitively depend 
on the chosen A as long as A is much greater than the im- 
purity scattering strength. Moreover, the spin-resolved 
FT-LDOS can be obtained if we separate each compo- 
nent i when evaluating function g(k, q, uj), i.e., i = 1 for 
spin-up and i = 2 for spin-down. 

In principle, for the case of an edge-impurity scat- 
tering, one can use Eqs. ([3])- ([5]) to compute the LDOS 
from 5p(r, uj)=-Im^2 i 5Gu(r, r, uj)/tt in a straightforward 
manner. However, it is more convenient, without loss of 
generality, to treat this scattering problem by using an 
analogy of the elementary scattering problem with a bar- 
rier potential in one dimension, which is directly based 
on the wave function point of view. Our method is briefly 
sketched in section III C. 



III. NUMERICAL RESULTS 

We compute the induced LDOS at selected uj, 
5p(q,uj), for the nonmagneic/magnetic impurity case, 
and, p(q x ,uj), for the edge impurity case. Our numerical 
results are reported for a representative potential scat- 
tering strength, Vni = Vmi — Vo =0.05eV. The cho- 
sen imaginary part of the energy r] =10meV has been 
checked to be insensitive to the observed main features. 



Also, in our analysis a 400 x 400 momentum grid is used 
in (— 7r, 7r) x (— 7r,7r) k space and 200 discrete points are 
displayed within (— tt, tt) along each direction in q space. 
Note that the relevant range of SBZ in experiments would 
correspond to about 5.5 times larger than 2tt. 
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FIG. 4: The spectral function A(k,oo) at uj = 0.3eV with 
three most possible scattering wave vectors. Note the wave 
vector is in units of 7rfr -1 and brighter region corresponds to 
higher spectral weight. 
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FIG. 5: The real part of the Fourier transform of local density 
of states in the case of single nonmagnetic point impurity at 
(a) uj = 0.3eV, and (b) uj = OmeV. 



A. Nonmagnetic point impurity 

We first consider the interference patterns in a 2D he- 
lical liquid with a nonmagnetic point impurity. Start- 
ing with oo = Ew =0.3eV far away from the Dirac point 
(uj = 0), the shape of the FS is now like a hexagram. This 
is just the energy range where experiment may achieve 
without subtle chemical tuning near the surface of a 3D 
topological insulator. As we will see later, such energy 
range indeed provide a better chance to reveal the topo- 
logical nature of the helical Fermion system. In Fig. 0J 
the spectral function, *A(k, uj) = — ^Im[TrGo(k, uj)] at 
uj = 0.3eV, are plotted with scattering vectors on top, 
which are expected to associate with high joint DOS on 
a constant-energy contour. 

As shown in Fig. [5] (a), the interference pattern 
includes six sharp peaks along YK outside a com- 
plicated, hexagon-shaped pattern centered at Y and 
other six weaker peaks along YM slightly inside the 
hexagon. These two sets of peaks simply correspond 
to (±qi3,±q 3 5,±q 5 i) and (±qi 2 , ±q23, ±Q34), respec- 
tively, as indicated in Fig. [H However, the most promi- 
nent feature we observed here is that those expected 
peaks, which correspond to the (±qi4, ±q25, ±q36)> are 
entirely absent. This apparent puzzle can be understood 
by the absence of backscattering between two time re- 
versal connected partners, as shown ini^. Suppose in our 



scattering problem, |k, j) is the incoming state, while its 
time-reversal partner, | — k, j) oc T|k, |), is the outgoing 
state. T is the time-reversal operator with the property 
T 2 = —1. For any time-reversal invariant and hermitian 
operator V (such as our nonmagnetic scattering poten- 
tial), we have 

(-MIV1M) = (T(k,t)|l/(k,T)) = (Tl/(k,T)|T 2 (k,T)) 
= -<k, t \rv\k, t>* = -<k, t \vr\k, t>* 
= -<k,T|y|-k,i>* = -(-k,||yt| k ,T> 

= -<-k,||V|k,T) = 0. ( 8 ) 

In other words, the backward scattering between time- 
reversal partners is not allowed. This naturally explains 
the absence of the interference peaks, corresponding to 
q36 (and of the same type). Such a behavior sharply 
distinguishes the 2D helical Fermion system from a con- 
ventional metal. In addition, it might be worth men- 
tioning here that the angles of our observed interference 
peaks, q35, appear different from the experiment done by 
Zhang et a/. 15 , where there exhibits six peaks along TM, 
instead of YK as displayed in Fig. [5] (a). We would like 
to postpone this issue to the discussion section. 

When further increasing the Fermi level, the vertices 
become sharper and the joint DOS at fixed q35, how- 
ever, is suppressed. As a result, the six peaks seen in 
Fig. [Sfa) diminish and the replaced feature turns out to 
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be the other six peaks at fixed q', corresponding to the 
scattering vectors connecting between second neighbor 
of the convex parts of the FS (see Fig. [6]), which were 
observed in recent experiments 15 . On the other hand, 
when the Fermi level gets closer to the Dirac point, for 
instance, uj =0.05eV, the interference pattern becomes 
almost isotropic with obvious stronger weight within a 
circular region, as shown in Fig. [5] (b). The size of the 
region can be estimated to be a disk with twice longer ra- 
dius of the corresponding circular FS of the system. This 
is basically consistent with our CCE picture (see Fig. [I]), 
where no finite, specific q vectors can be picked out when 
ou approaches to the Dirac point. 
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FIG. 6: The real part of the Fourier transform of local density 
of states in the case of single nonmagnetic point impurity at 
uj = 0.375eV. 



sity of the system, namely, instead of having (q, oj) = 
£/?j(q, uj) as in the nonmagnetic impurity case, we have 
£/?j(q, uj) ~ — #P|(q, oj) . This effect can be easily un- 
derstood. Suppose we are considering an impurity mo- 
ment along the z-direction, then the spin-up electrons 
and spin-down electrons see two scattering potentials of 
opposite signs. In the lowest order of perturbation the- 
ory, the scattering amplitude of the spin-up and spin- 
down electrons thus differ by a minus sign so that the to- 
tal interference pattern of the charge density vanishes al- 
most everywhere. The same argument no longer holds if 
higher orders of perturbation are included. For the model 

x10~ 3 
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B. Classical magnetic point impurity 

Next, we study the QPI induced by a time-reversal 
symmetry breaker, a magnetic impurity^i. We focus on 
the effects of a classical magnetic impurity so that the 
Kondo physics is ignored. In the following, after describ- 
ing general features of the QPI with a magnetic impurity, 
we will discuss the cases separately when the impurity 
moment is fixed along x, y, and z directions. 

Different from nonmagnetic impurities, a weak mag- 
netic impurity has very little effect on the charge den- 



FIG. 7: The real part of the Fourier transform of charge local 
density of states in the case of single magnetic point impurity 
with its spin polarized along the z-axis at (a) uj = 0.3eV, and 
(b) uj = OmeV. 



considered here, we can explicitly prove the above state- 
ment. Assuming V <C cj, the approximation T(uo) w V 
becomes sufficiently accurate. In this case (impurity mo- 
ment along ^-direction), we have 



Tr[6G(q,i 



d 2 k 
(2tt)2 



TV[G (k,w)l/Go(k + q, 



V 

V 
0. 



d 2 k Tr[(wo- -fc 

+ + k 3 _)G z )G z (ujGQ - (k y + q y )a x + (k x + q x )a y + |((fe + q)\ + (k + q)-)<T z 

((uo + irj) 2 - 4(k))((a; + iv) 2 ~ 4(k + q)) 
d 2 k k\ + k 3 _ + (k + g)\ + (k + qf_ + ik y (k x + q x ) - ik x (k y + q y ) 



(27T) 2 



(( W + iv )2 _ 4 (k))((w + ir,f - e 2 + (k + q)) 



(9) 
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The last equality is achieved by shifting the origin to 
(Qx •>%)•> changing the integrated variables k to — k, and 
taking the advantage that e+(k) = e+(— k). Similar 
derivations hold for the impurity moment along x and 
^/-directions. If the second order term 0(V 2 ) is included 
in the T-matrix, the cancellation becomes no longer valid, 
and there is indeed small but finite charge LDOS pattern 
in the system. In Fig. [71 we plot the numerical results 
of £p(q, uj) at uj = 0.05,0.3. It is clear that The ampli- 
tude of charge density variation by magnetic impurities 
in Fig. is two orders of magnitude smaller than that 
shown in Fig. [5] by nonmagnetic impurities. 

Therefore, for the magnetic impurity case, we should 
choose a time-reversal breaking observable to study the 
interference, and a natural choice is the spin local density 
of states (SLDOS), defined by 

S(r,u) = -^Im[ J dt6(t)(c a (r,t)a^cl(r,0))e^%(10) 

where c^(r, t) creates an electron with spin polarization 
a at position r and time t. From now on we will only 
focus on the FT of the ^-component SLDOS. 
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FIG. 8: The real part of the Fourier transform of spin local 
density of states in the case of single magnetic point impurity 
with its spin polarized along the z-axis at (a) uj — 0.3eV, and 
(b) uj = OmeV. 



In the case of nonmagnetic impurity, we have demon- 
strated the absence of interference between |k, f) and 
|— k, |), which form a time-reversal pair. Physically, a 
time-reversal breaker such as a magnetic impurity can 



lift this ban on the backscattering. Similar to Eq. ([8]), 
it is easy to show that (— k, j |V"|k, |) ^ due to 
T<TiT~ l = —(Ti. This feature is universal in all of our 
figures for magnetic impurity. Taking Fig. [8ja) as an 
example, we can compare it with Fig. [3(a) and notice 
that although they have common features, the points in 
the FT-SLDOS that associate with the 2kp backscatter- 
ing scattering vectors is only present (±qu, ±q25, ±q36) 
(see Fig. H]) in the magnetic scattering. We can also com- 
pare Fig. [H^b) for magnetic scattering with Fig. ^h) for 
nonmagnetic scattering when uj = 0.05eV. In the latter 
case, the interference strength universally decays quickly 
after reaching the boundary of the circle; while in the 
former case, the interference strength reaches a negative 
peak across the boundary, indicating a scattering that 
flips spin moments of the quasiparticles. 

a i.o 
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FIG. 9: The (a) real part and the (b) imaginary part of the 
Fourier transform of spin local density of states in the case of 
single magnetic point impurity with its spin polarized along 
the y-axis at uj = 0.3eV. 



Now, we discuss the QPI by magnetic impurities with 
in-plane magnetic moments. In this case, a unique fea- 
ture rises in the FT-SLDOS. As shown in Fig. (9^ind in 
Fig- HOI at uj = 0.3 we plot two figures, which correspond 
to the real and imaginary parts of the FT-SLDOS sepa- 
rately. Similar to the LDOS, the real and imaginary parts 
correspond to the symmetric and antisymmetric parts of 
S z (x, y, t) respectively. For magnetic impurity with mag- 
netic moment along z axis, the symmetric part dominates 
and the antisymmetric part is either vanishing or orders 
of magnitude smaller than the symmetric part. However, 
here as shown in Fig. [9l the antisymmetric part is about 
three times larger than the symmetric part, the result 
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can be understood as follows. An inversion transforma- 
tion in a two dimensional plane, i.e. (x,y) — > (—x,—y) 
takes a z (x,y,t) — > cr z (—x,—y,t) and (T x ,y(—x,—y,i) — > 
—&x,y{—%,—y,t)- Therefore, under this transformation, 
the Hamiltonian without the warping term in the pres- 
ence of magnetic impurities with in-plane magnetic mo- 
ments transforms as H(Vq) — > H(—Vq), where Vb is the 
coupling strength of magnetic impurity. Thus, from this 
symmetry, if we consider S z (x,y,t) as function of Vo as 
well, we have S z (x, y, t, Vb) = S z (— x, —y, t, — Vb). There- 
fore, the first order correction from the scattering poten- 
tial vanishes for the symmetry part. In the presence of 
the warping term, there is no such an exact symmetry ar- 
gument. Nevertheless, the symmetric part is still much 
smaller than the antisymmetric part. In the following, 
we will first focus on the antisymmetric part. 

Fig. [51(b) shows the (antisymmetric) FT-interference 
pattern for the impurity moment along the ?/-axis at 
uo = 0.3eV. We find that the strongest interference ap- 
pears at wave vector ±qsi in Fig. [51(b) is defined in 
Fig. Moreover, qi3 and q35 do not present as strong 
peaks, in contrast with the cases of the nonmagnetic im- 
purity and the magnetic impurity spin along z-axis. In 
addition, a remarkable feature in the interference pat- 
tern is that S^(q, w) is zero on the line q y = 0. This 
is caused by an exact symmetry of the system which 
dictates S z (x,y,t) = —S z (x,—y,t). This point will be 
discussed later in length. Fig. fTOT b) shows the (anti- 
symmetric) FT-interference pattern for the impurity spin 
along the x-axis at uo = 0.3eV. We can see that the 
strongest interference is associated with the vertex-to- 
vertex wave vectors qi3 and q35. The strong peak at 
q5i does not appear and we have S^(0,q y ,(j) vanish- 
ing. This result stems from an approximate equality 
S z (x,y,t) w S z (x, —y,i), a point of which will be dis- 
cussed next. 

We can understand above detailed features in the SL- 
DOS from the symmetry analysis of the model. The 
model obviously has the time-reversal symmetry and the 
three- fold rotation symmetry. Moreover, the model also 
preserves the y — > —y mirror symmetry (m y ) but breaks 
the x — > — x mirror symmetry (m x ), as can be seen in the 
warping term. Explicitly, the m x operator takes k± to 
k T and <j z to — a z , which changes the sign of the warping 
term. Now, let us consider the system in the presence 
of a magnetic impurity with its spin along 2/- axis. Since 
s y — > Sy under m y , the whole system still preserves the 
mirror symmetry m y . This symmetry directly leads to 

S z (x,y,v) = -S z (x,-y,u). (11) 

This symmetry property is clearly demonstrated in 
Figs. [51(a) and (b). On the other hand, if the impu- 
rity spin is fixed along the x-direction, the system does 
NOT have m x symmetry and we have S z (x,y,uo) ^ 
—S z (—x,y,uo). This feature is also demonstrated in 
Fig. [Tola). If we had S z (x,y,uo) = —S z (—x,y,uo), we 

should have S z ^ (q x , q y , uo) = -S z ^ S \-q x ,q y ,uo) or 
S z (x,y,uo = S z (—x,y,uo) = 0. However, in Fig. HOlf a), 




-1.0 -0.5 0.5 1.0 

q x /(*b" 1 ) 




-1.0 -0.5 0.5 1.0 

q x /(*b" 1 ) 



FIG. 10: The (a) real part and the (b) imaginary part of the 
Fourier transform of spin local density of states in the case of 
single magnetic point impurity with its spin polarized along 
the x-axis at uo = 0.3eV. 



it is clear that Sf (q x ,q y ,(jj) = (—q xi q yi uj) 7^ 0. 

The above symmetry is a very important property of 
the model. In fact, to simply account for the shape of FS, 
we may also artificially make the Fermi velocity strongly 
angle dependent while keeping the same spin texture 
where all spins on the FS are in-plane without tilting. 
For instance, we can write 

k 2 

H(k) = v(k)(k x a y - k y a x ) + — , (12) 



where v(k) = y v 2 + \ 2 k A sin 2 (3#), with 6 being the az- 
imuthal angle with respect to x axis (TM). This model 
(the in-plane model) has the same dispersion as the 
model in EqUJ but has only in-plane spin texture. The 
symmetries of the SLDOS here can help us distinguish 
these two models. For example, one can check these two 
equations experimentally: S z (x,y,oj) = —S z (x,—y,oj) 
for impurity spin polarized along y axis and S z (x, y, uo) = 
—S z (—x,y,uo) for impurity spin polarized along x axis. 
If both are held, then the in-plane model suffices; but if 
only one is held, we may need an out of plane spin (warp- 
ing) term. In TableHl we list the property of SLDOS in 
the two models, Eq.([T]) and Eq. (p!2|) . in the presence of 
different types of impurities and under basic symmetry 
operations. 
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(a) 



s z 


m x 


TTly 


c 3 




s z 


Vflx 


TTly 


c 3 


Sx 


X 


« 1 


X 






-1 


« 1 


X 


Sy 


X 


-1 


X 




Sy 


1 


-1 


X 


S z 


& 1 


1 


1 




Sz 


1 


1 


1 



(b) 



TABLE I: The symmetry of S z {x,y,t) under symmetry oper- 
ations of mirror-x (m x ), mirror-y (m y ) and three- fold rotation 
about z-axis (C3) with impurity spin along three axes, (a) is 
for the model in Eq.(pQ) and (b) the model in Eq. ([T2]) . '1' 
means symmetric; ' — 1' means antisymmetric and 'x' means 
neither of the above. The means it is symmetric (anti- 
symmetric) in the weak impurity strength approximation. 



C. Nonmagnetic edge impurity 

Step atomic roughness on a surface may be locally ide- 
alized into an edge impurity, that is, an infinite line with 
different but uniform potential on two sides. An edge im- 
purity in a 2D conventional Fermi gas is known to give 
rise to Friedel oscillation at fixed energy in the LDOS. 
This oscillation can simply be understood as an interfer- 
ence pattern between the incoming plane wave and the 
reflected wave by the ID edge. The major contribution 
comes from the two opposite k-points on the constant en- 
ergy contour, ±kp, and the oscillation has the wavenum- 
ber 2|kp| while decaying as a form \j\fd where d is the 
distance from the edge impurity^ 2 -. The same picture is 
no longer valid if the state at k and — k do not scatter 
with each other, a case for the surface states of a 3D topo- 
logical insulator where the backscattering is forbidden by 
the time-reversal symmetry. Therefore the oscillation is 
expected to decay much faster and thus practically ab- 
sent in an STM experiment. The 'absence' of the Friedel 
oscillation is considered as a sign of (spin) helical Dirac 
Fermion systems. However, the oscillation has been ob- 
served in STM experiments 16 . The apparent discrepancy 
between theory and experiment was soon claimed to be 
superficial and explained by the hexagram-shape of the 
FSU^. In this subsection an exact calculation is performed 
to test this physical picture. 

We consider that the edge impurity is fixed along y 
axis and the system has zero potential for x < and 
uniform potential V for x > 0. A general quantum state 
on the left hand side (LHS) takes the form 



^{kxj k y ] Xj y) — 
and the LDOS is 



</>o(k x , ky] x, y) + r</> (-k x , k y ] x, y) 



(13) 



I 

Jk x >0 



d 2 k 

-—-^\^(k x ,k y ]x,y)\ 2 5{uj - e+(k x ,k y )). 
>o (2<T 

(14) 

The reflection amplitude r can be obtained together with 
the transmission amplitude t by matching the boundary 



condition at the edge, namely, 

<fo(k x ,ky',0,y) + r<f>o(-k x ,k y \0,y) = £0o(^"> k y \ 0, y), 

(15) 

where k'J. is fixed by the energy conservation e(h Xl h y ) = 
e(K,k y )-V. 



a 0.03 



0.02 



0.01 







— \f 



0.5 , 

q /(Tib -1 ) 




FIG. 11: (a) The Fourier transform of the edge impurity (V — 
—0.1) interference pattern. (b)The three k's that dominate 
the interference pattern on the energy contour at uj = 0.5eV. 



Fig.dlja) shows the FT-LDOS for the LHS of the edge 
impurity at uo = 0.5. We can clearly identify the two 
peaks in the interference associated with q x = 2k2 and 
q x = 2k3, defined in Fig. ITTlf b). No feature is present at 
q x = 2ki , reflecting the absence of backscattering. The 
spatial dependence of the oscillation, a real space LDOS, 
is given in Fig.[T2lfa). A clear beating pattern can be seen 
with spatial period ~ (k3 — k2) _1 . The oscillation decays 
like l/|x| a where a ~ 0.46, qualitatively matching the 
theoretical prediction in the large \x\ limi t 22 ! 23 . When 
\x\ is large enough, the stationary points approximation 
tells us that, if the edge impurity is along the y- axis, the 
interference pattern is dominated by the k-points where 
k x reaches local minimum or maximum. In our model, 
k 2 (3) are the points corresponding to the minimum (max- 
imum) of k x on the contour of constant energy. However, 
The existence of such extrema depends on co. If uo is 
small enough, the extrema k2,3 disappear and we are left 
with only ki . Since ki is not allowed to scatter with its 
time-reversal partner, the decaying of Friedel oscillation 
becomes |x| -3 / 2 at large \x\ [see Fig. [T2l b)]. Therefore, 
there is no universal function for the oscillation decay. 
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The decay depends on the values of parameters. T here 
are two inherent length scales in the model: b = \J X/v 
and b' = v/lu. If b > 1.486', the energy contour is a hex- 
agram and an l/y/jx| decay of the oscillation appears, 
while if b <C we have a nearly circular FS and the 
decay of oscillation takes the form p(x) ~ |x| -3 / 2 . In the 
intermediate range, the oscillation varies. For example, 
at b = 1.2b' {uj = 0.3), the oscillation decays exponen- 
tially for \x\ < 1006 but close to \x\~ 3 ^ for \x\ > 2006. 



a o.i 




-300 -200 -100 

Position 



b 

0.04 



0.02 




-0.04 ^ 

-200 -100 

Position 

FIG. 12: The real space interference pattern for the edge 
impurity (V = -0.1) at (a) uj = 0.5eV and (b) uj = 0.05eV. 
The density fluctuation Sp is denned as 5p = p — po = p — 1. 
The position x is in units of b. 



IV. DISCUSSION AND CONCLUSION 

The model we have solved produces interference pat- 
terns that have enough features to be associated with 
the topology-protected surface states and the effects of 
the hexagonal warping term in 3D topological insulators. 
However, in order to be more careful before making con- 
clusions, there are two more remarks we would like to 
mention here. 

(i) In our calculations, we neglected the possibility of 
any ordering due to interaction-induced FS instability. 
This is valid as long as there is no significant FS nesting 
vector—. In addition, we do not expect strong electron- 
electron interaction based on the following observation. 




exp(-x/d) 

1/x 

data 



50 100 150 

X(A) 

FIG. 13: Fitting the experimental data of RefHiSl using dif- 
ferent oscillating functions. The experimental energy -62meV 
corresponds to ~ 0.25eV in our units. In the exponential fit, 
d = 107 A. 



In experiments on topological insulators, the Fermi level 
of the sample in general is closer to the bottom of the con- 
duction band and is far away from the Dirac point. Such 
a system with finite density of states may provide enough 
screening effect to Coulomb interaction between surface 
electrons. Moreover, attempting to tune the Fermi level 
lower by a metallic gate may also lead to the same phe- 
nomenon, turning interaction between electrons into ir- 
relevant regime. 

(ii) In real systems, there is no 'purely magnetic' im- 
purity. A magnetic impurity should also have a non- 
magnetic component. This fact does not change our re- 
sults obtained for magnetic impurities. In the parameter 
region we choose, the weak impurity approximation is 
always valid (see a detailed discussion of this approxima- 
tion in the Appendix), the non-magnetic impurity only 
leads to the charge density modulation and has little ef- 
fect on the SLDOS. Namely, the magnetic part of impu- 
rity is solely responsible for the SLDOS. 

(iii) As we noticed in section III A, the STM experi- 
ment done by Zhang et alJ£ on [111] surface of Bi2Te3 
exhibited six peaks in FT-LDOS for the case of nonmag- 
netic impurities. The experimental result differs from 
our results shown in Fig. [5ja) by a 30 degrees of rotation. 
However, this discrepancy can be understood by noticing 
that in the energy range where they observed the clear 
interference patterns (50meV~400meV), the surface den- 
sity of states are mixed with bulk states along YM . Con- 
sequently, due to the superposition of waves with various 
wavelengths the interference patterns are simply smeared 
out in these regions. Instead of a full FS we considered 
here, the dominant interference patterns are then from 
other unmixed parts of the FS, ie., the parts along TK. 

(iv) In an STM experiment done by Alpichshev et 
al^, the decaying behavior of the Friedel oscillation was 
claimed to be l/\x\. However, in the case of ID edge 
impurities, our calculation shows l/l^ 1 / 2 behavior if the 
FS shape is dominated by the warping term, and |x| -3 / 2 
if the warping term is negligible. We believe there are 
two possible sources of the discrepancy. First, we notice 
that a simple fitting to the first several periods of oscil- 
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lation is not enough to determine the decaying behavior. 
In Fig fT3| we show that the data in RefQjJ can also be 
well fitted using an exponentially decaying; function, as 
opposed to the l/|x|-type fit used in Reflld Second, the 
experimental measurements are not a pure surface effect. 
There are bulk electrons in the nearby conduction band 
which can cause different decaying behavior and compli- 
cate the issue. More future experimental measurements 
are necessary to resolve the issue and test the theoretical 
predictions. 

(v) We also notice that a similar theoretical work24 fo- 
cusing solely on nonmagnetic impurity was posted online 
recently, which suggests the six peaks at fixed q's that 
correspond to the scattering vectors connecting between 
second neighbor of the convex parts of the FS dominate 
in the QPI patterns. Their results are consistent with 
our calculations since their results, according to the en- 
ergy unit in our paper, are obtained at uo = 0.375eV. 
However, our results suggest that the relative strength 
between the interference at q's connecting next near- 
est neighboring vertices (e.g., q35 and the interference 
at q's connecting next nearest neighboring arc-centers 
(e.g., q2'4' in the QPI patterns is quite subtle and de- 
pends on energy. Therefore, a full T matrix calculation 
is necessary in calculating the QPI patterns. 



In conclusion, we have investigated the quasiparticle 
scattering in a 2D helical liquid in the presence of non- 
magnetic/magnetic point impurity or an nonmagnetic 
edge impurity. The inclusion of the hexagonal warping 
term in our system not only inherits the nature of the k- 
linear helical liquid but also sharpens our features men- 
tioned above by distorting the shape of the FS. More im- 
portantly, it requires an out of plane spin texture and can 
be distinguished from other systems with examination of 
the mirror symmetries when the magnetic point impurity 
with in-plane spin moment is present. The absence (pres- 
ence) of spots in FT-LDOS (FT-SLDOS), corresponding 
to the backscattering interference, are the essential fea- 
tures to confirm the topological nature of the helical liq- 
uid. The results in our work, as may be detected by STM 
experiments, can be a useful quantum signature, which 
is uniquely associated with this new phase of matter, a 
3D topological insulator. 
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APPENDIX A: THE WEAK IMPURITY 
STRENGTH APPROXIMATION 

For the parameter we used throughout the paper, the 
scattering strength is relatively small (i.e., Vop(co) < 1). 
In this limit the approximation T(ui) ~ V is consider- 
ably accurate (less than 3% error in our case), and many 
approximate equalities may be derived thereof. This sub- 
section is devoted to explicitly deriving these relations. 

First we prove that for a purely magnetic impurity, 
the induced (charge) LDOS is almost zero everywhere. 
We prove this by showing Tr[£G(q, uj)] « 0. Within the 
approximation, we have 



Ti[5G(q,uj) 



4^ 



Tr[G(k,^)T>G(k + q,^) 



/d 2 k 
-^Tv[G(k,Lo)aiG(k + q, 



(Al) 



In the equation above we do not specify the spin polar- 
ization of the impurity and the result is general. Noticing 



that the system is invariant under time reversal operation 
(C = ia y ), i.e., CH^C' 1 = # T (-k) and that a mag- 
netic impurity changes sign under the same operation, 
i.e., Cg{C~ y = — <7 T , we have 



d 2 k 
4^ 



Tr[G (k,^)a,G (k + q, 



/ 

= ^Tr[CG (k )W )(T i Go(k + 6 9)W )C- 1 ] 
= -|0Tr[Gj(-k,cK^(-k-q, w ) 

/d 2 k 
^2 Tr[G (-k - q, cj)cr;G (-k, uj)] 

= 0. 



(A2) 



The last equality may be understood after changing vari- 
ables k — > — k — q. 

Next we show that the approximate symmetries listed 
in Tabl^a) hold within the same approximation. Ac- 
cording to the table, we have, for impurity spin (again 
it is a purely magnetic impurity) along the z-axis, the 
SLDOS S z (x,y,i) w S z (—x,y,i), which is equivalent to 
S z (q x1 q y ,uo) w S z (—q xi q yi uj). This may be derived in 
the following way: 



S z (q,L0)=Tr[5G(q,uj)a z ]n J -0^Tr[G o (k,co)VG o (k + q,co)a z } 

d 2 k Tr[(w<7o - k y a x + k x a y + ±{k\ + k 3 _)a z )a z (uja - (k y + q y )a x + (k x + q x )a y + + q)\ + (k + q) 3 _)<r z 
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(2tt)2 ((uj + irj)* - e* (k))((a; + irj)* - e\(k + q)) 

V f (2 U2 + ky< " ky + % ^ + kx ^ x + ^ + ^+ + k ~^ k + ^+ + V* - Q ^ ) 

J (2^) 2 ( ((uj + iv) 2 - 4( k ))((" + ^) 2 - 4( k + q)) 

« S z (-q x ,q y ,uj). (A3) 

I 

In deriving the last equality, we notice that e(k x ,k y ) — for the impurity spin in x and z directions. Since 
e(—k x , k y ) and change variables as k x —> —k x . In the sec- <j y H(k x , k y )a y = H(k x , —k y ), we have 
ond column of Tabl^Ja), we find S z (x, y, t) w S z (x, —y, t) 



S z (q x ,q y ,u) 



f d 2 k 

/ ^Tr[Go(k x ,k y ,uj)cr x , z Go(k x + q x ,k y + q y ,uj)cr z ] 

/d 2 k 
-^Tr[Go(k x ,-k y ,u)(-<T XiZ )Go(k x + q x ,-k y - q y ,uj)(-a z )] 

/d 2 k 
-^Tr[G (k x ,k y ,uj)(j x , z Go(k x +q x ,k y -q y: uj)a z ] 



S z (q x , -q y ,v). 



(A4) 



A simple consequence of the weak impurity approxima- tion is a linear combination of LDOS or SLDOS when 
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there is more than one impurity, or an impurity that 
has both magnetic and non-magnetic parts. In the lat- 
ter case, one can simply add up the FT-LDOS and FT- 
SLDOS for each part to obtain the total configuration. 
But as discussed in the text, the magnetic part con- 
tributes very little to the LDOS, and the non- magnetic 
part does not contribute to the SLDOS (obvious from 
time-reversal symmetry) , most of the results for the mag- 
netic impurity part remain the same. 



For the refraction part, <j>" is fixed by 

^ Fq sin^") = fcsin(0). (B4) 

And for the reflection part, (j)' = ir — <fi. Solving these 
equations, one has 



APPENDIX B: FRIEDEL OSCILLATION AT 
FIXED ENERGY IN A 2D DIRAC METAL BY 
AN EDGE IMPURITY 

In the text, we stated that when the energy lies within 
the 'Dirac regime' (e.g., when uo = 0.05eV), the decay of 

the Friedel oscillation takes the form p(x,oj) oc |x| -3 ^ 2 . 
In this subsection the asymptotic expression for the 
LDOS oscillation caused by an edge impurity in a 2D 
Dirac metal is derived. The Hamiltonian takes the form 



H(k) =vk-a. 



(Bl) 



This form is equivalent to the linear part in Eq(pQ) up to a 
global spin-SU(2) gauge. The Hamiltonian may be easily 
solved: (only positive energy solutions are listed) 



e(k) = vk, 
^(k) = (e^ /2 ,e- 



(B2) 



where (j) being the polar angle. Now let us suppose that 
the space is divided in half at x = (i.e., the edge im- 
purity is along y-axis), and the right side has a uniform 
potential of V = — Vo where Vo > 0. The continuity of 
the wavefunction at x = gives 



r(4>) 



e icf>'/2 
e -i<f>'/2 



t{4>) 



Jcf>"/2 
-i<j>" /2 



<B3) 



sin(^) 



<P+<4>" ■ 



(B5) 



cos( 
cos((/>) 
cos(^) 



Using the stationary phase approximation, we know 
that after integrating all the k's on the fixed energy con- 
tour uo = vk to obtain the LDOS, the contribution mainly 
comes from the k's that have small polar angles. At small 
angles, the reflection index takes the form 



r{4>) w - 
Using Eq(p3j), we have 



Vo 



2{V + vk) 



(B6) 



d4> V m2 . 2w ... 
p(x,u>) w / T^-TT- — <t> ™ — cos{(j>)x) 

27T Vo + UJ V 



Vo 



L^VoT^ Im[4> 

-^^co S (^-J)(^)- 3 / 2 (B7) 
Vo+ou 4 v v 4 yv v J vy 



2 p i^f- cos(0)cci 



